{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 用户社交数据（user_friends.csv）处理\n",
    "（只取训练集和测试集中出现的用户ID）\n",
    "\n",
    "数据来源于Kaggle竞赛：Event Recommendation Engine Challenge，根据\n",
    "events they’ve responded to in the past\n",
    "user demographic information\n",
    "what events they’ve seen and clicked on in our app\n",
    "用户对某个活动是否感兴趣\n",
    "\n",
    "竞赛官网：\n",
    "https://www.kaggle.com/c/event-recommendation-engine-challenge/data\n",
    "\n",
    "user_friends.csv文件：共2维特征\n",
    "user：用户ID\n",
    "friends：以空格隔开的用户好友ID列表"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 导入工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "import numpy as np\n",
    "import scipy.sparse as ss\n",
    "import scipy.io as sio\n",
    "\n",
    "#保存数据\n",
    "import pickle\n",
    "\n",
    "from sklearn.preprocessing import normalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总的用户数目超过训练集和测试集中的用户，\n",
    "为节省处理时间和内存，先去处理train和test，得到竞赛需要用到的活动和用户\n",
    "然后对在训练集和测试集中出现过的事件和用户建立新的ID索引\n",
    "先运行user_event.ipynb,\n",
    "得到事件列表文件：PE_userIndex.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 读取之前算好的测试集和训练集中出现过的用户"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of users in train & test :3391\n"
     ]
    }
   ],
   "source": [
    "#读取训练集和测试集中出现过的事件列表\n",
    "userIndex = pickle.load(open(\"PE_userIndex.pkl\", 'rb'))\n",
    "n_users = len(userIndex)\n",
    "\n",
    "print(\"number of users in train & test :%d\" % n_users)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 读取之前用户-活动分数矩阵，将朋友参加活动的影响扩展到用户"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#用户-事件关系矩阵\n",
    "userEventScores = sio.mmread(\"PE_userEventScores\")\n",
    "\n",
    "#后续用于将用户朋友参加的活动影响到用户\n",
    "eventsForUser = pickle.load(open(\"PE_eventsForUser.pkl\", 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# user_friends.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#读取数据\n",
    "\n",
    "\"\"\"\n",
    "  找出某用户的那些朋友\n",
    "  1)如果你有更多的朋友，可能你性格外向，更容易参加各种活动\n",
    "  2)如果你朋友会参加某个活动，可能你也会跟随去参加一下\n",
    "\"\"\"\n",
    " \n",
    "#用户有多少个朋友\n",
    "numFriends = np.zeros((n_users))\n",
    "userFriends = ss.dok_matrix((n_users, n_users))\n",
    "    \n",
    "fin = open(\"user_friends.csv\", 'rb')\n",
    "#字段：user，friends\n",
    "fin.readline()                # skip header\n",
    "\n",
    "#ln = 0\n",
    "for line in fin:  #对每个用户        \n",
    "    cols = line.strip().decode().split(\",\")\n",
    "    user = str(cols[0])    #user\n",
    "    \n",
    "    if user in userIndex:   #该用户在训练集和测试集的用户列表中\n",
    "        friends = cols[1].split(\" \")  #friends\n",
    "        i = userIndex[user]       #该用户的索引\n",
    "        numFriends[i] = len(friends)\n",
    "        for friend in friends:  #该用户的每个朋友\n",
    "            str_friend = str(friend)\n",
    "            if str_friend in userIndex:  #如果朋友也在训练集或测试集中出现\n",
    "                j = userIndex[str_friend]   #朋友的索引\n",
    "            \n",
    "                # the objective of this score is to infer the degree to\n",
    "                # and direction in which this friend will influence the\n",
    "                # user's decision, so we sum the user/event score for\n",
    "                # this user across all training events.\n",
    "            \n",
    "                #userEventScores为用户对活动的打分（interested - not interseted）\n",
    "                #在Users-Events.ipynb中计算好了\n",
    "                eventsForUser = userEventScores.getrow(j).todense()\n",
    "            \n",
    "                #所有朋友参加活动的数量（平均频率）\n",
    "                score = eventsForUser.sum() / np.shape(eventsForUser)[1]\n",
    "                userFriends[i, j] += score\n",
    "                userFriends[i, j] += score\n",
    "            \n",
    "fin.close()\n",
    "    \n",
    "\n",
    "#用户的朋友数目\n",
    "# 归一化数组\n",
    "sumNumFriends = numFriends.sum(axis=0)\n",
    "numFriends = numFriends / sumNumFriends\n",
    "sio.mmwrite(\"UF_numFriends\", np.matrix(numFriends))\n",
    "\n",
    "#\n",
    "userFriends = normalize(userFriends, norm=\"l2\", axis=0, copy=False)\n",
    "sio.mmwrite(\"UF_userFriends\", userFriends)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.83273440e-04, 6.57934055e-04, 5.54754987e-05, ...,\n",
       "       4.68995762e-04, 2.73893525e-04, 3.76268600e-04])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numFriends"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
